fix: align and persist remote embedding result metadata#6
Open
fix: align and persist remote embedding result metadata#6
Conversation
Replace n_tokens_truncated with a shared truncated flag across local, remote, and custom embedding results. Keep remote token counts useful by preferring exact and estimated prompt token metadata when present. This avoids carrying truncation semantics that were exact for the local engine but misleading for remote vectors.space responses.
Read output_dimension, data[0].embedding, data[0].truncated, and nested usage token fields from the documented vectors.space response shape. This makes the remote embedding parser match the current API envelope instead of relying on flat key scans across the whole JSON payload.
Align the public C API reference with the current dbmem_embedding_result_t shape by documenting the truncated boolean flag instead of n_tokens_truncated. This avoids publishing stale custom-provider docs after the shared embedding result metadata was simplified across local, remote, and custom engines.
- add schema versioning and migrate vault/cache metadata columns - store n_tokens and truncated on vault rows and cache entries - assert metadata persistence in unittest coverage - document the remote parser refactor plan for later implementation
…gure Add four e2e tests covering long-text behaviour and provider boundaries: - memory_add_long_text_chunking + memory_search_long_text_sections: structural and retrieval coverage for multi-chunk inputs. - memory_search_under_token_limit (~5KB, single chunk; passes) and memory_search_truncation_signature / _near_model_context (~10KB / ~19.5KB, rejected by provider): document vectors.space's 1024-token batch ceiling. Also: RUN_TEST no longer prints PASSED after a test fails, and Makefile strips LDFLAGS/CPPFLAGS/CFLAGS/LIBS from curl's ./configure so rebuilds work in shells with Homebrew-style env exports. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface n_tokens and truncated from dbmem_vault in the test output for the four search tests: - memory_search_long_text_sections: aggregate (chunk count, min n_tokens, any-truncated) for the chunked corpus. - memory_search_under_token_limit, _truncation_signature, _truncation_near_model_context: per-chunk values for both the short reference and the long chunk alongside ranking/score. Also append a per-test trailing sentence to the SHORT_REF in the two truncation tests so memory_add_text's content-hash idempotency doesn't collapse them into no-ops of the under-limit test's identical SHORT_REF. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
remote-embedding-parser-refactor-plan.md was committed by mistake — it was a local-only planning note. Remove it from tracking via git rm --cached so the working copy is preserved for the author while the file is dropped from the published tree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add example queries for the schema-version-2 columns on dbmem_vault so readers can discover token-usage aggregation and truncation diagnostics without having to read the source. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Embedding result metadata, unified and persisted. Replace
n_tokens_truncatedwith a sharedtruncatedboolean across local,remote, and custom embedding engines. Add
n_tokensandtruncatedcolumns to
dbmem_vaultanddbmem_cache, with schema versioning andmigration so existing databases pick them up cleanly.
Remote parser fixed. Read the documented vectors.space envelope
(
output_dimension,data[0].embedding,data[0].truncated, nestedusagetoken fields) instead of relying on a flat key scan. Prefer theexact / estimated prompt-token counts when present so
n_tokensismeaningful for cache and reporting use.
End-to-end coverage for the long-text path. New e2e tests cover
multi-chunk ingestion + per-section retrieval, single-chunk ingestion
under and over the provider's per-request token ceiling, and a single
chunk near the model context window. Each search test prints the
per-chunk
n_tokens/truncatedvalues it just round-tripped throughthe API, so regressions in the metadata path show up in the test log.
Build/CI hygiene. Run the SQLite extension unit tests in CI, fix
them under local-only builds, use portable temp paths in the sync
tests, hash the sync fixture from disk, and isolate curl's
./configurefrom inherited shell
LDFLAGS/CPPFLAGS/CFLAGS/LIBSso curlrebuilds work in environments with Homebrew-style env exports.
Docs aligned. Update the public C API reference for the new
truncatedflag on custom provider results.Test plan
make unittestpasses — covers schema versioning, migration ofpre-existing vault/cache rows, and persistence of
n_tokens/truncatedend-to-end.make e2e APIKEY=<vectors.space key>passes — exercises theremote parser and the persisted metadata against the live API.
local-only build variants.
memory_add_textcall, and confirm the new columns are populatedwithout errors.
three boundary tests (
memory_search_under_token_limit,_truncation_signature,_truncation_near_model_context) printtrunc=0for the under-limit case andtrunc=1for the twooversized cases — i.e. truncation is now observable as data
rather than as a 500.